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Summary 


The  purpose  of  this  research  was  to  conduct  a  field  test  of 
the  methodology  for  generating  efficiency  and  effectiveness 
measures  (MGEEM).  The  objectives  were  to  evaluate:  (a)  the 
number  and  types  of  measures  developed;  (b)  the  acceptability  of 
the  process  and  its  results  to  participants  (e.g.,  unit 
commanders);  (c)  the  cost  effectiveness  of  the  measures  in  terms 
of  their  use  of  existing  data;  and  (d)  the  extent  to  which 
independent  applications  of  the  methodology  to  similar 
organizations  produced  similar  sets  of  measures. 

Three  functional  areas  --  Administration,  Propulsion 
Maintenance,  and  Weather  --  were  selected  for  this  field  test. 
Eight  separate  organizations  in  each  of  the  three  functions 
served  as  the  test  sites.  For  each  organization,  a  prioritized 
list  of  Key  Result  A'-eas  (KRAs)  --  categories  of  results  which 
the  organization  is  expected  to  accomplish  were  developed, 
alorg  with  quantitative  efficiency  and  effectiveness  indicators 
for  each  KRA.  Implementation  of  the  MGEEM  involved  four 
activities;  (1)  development  of  a  systems  diagram  showing  the 
inputs,  outputs,  organizational  structure  and  required  contacts 
wa th  other  organizations;  (2)  establishment  of  a  management  team 
consisting  of  the  unit  commander  and  his/her  subordinates  to 
develop  KRAs  for  their  organization  by  means  of  the  Nominal  Group 
Technique;  (3)  establishment  of  a  second  team  to  develop  and 
prioritize  a  list  of  quan+itative  indicators  for  each  of  the  KRAs 
developed  by  the  first  team;  and  finally  (4)  review  of  the  list 
of  indicators  by  the  unit  commander  who  could  modify  or  delete 
indicators  judged  not  important  or  not  feasible  to  measure. 
Following  these  activities,  participants  from  each  team  were 
asked  to  complete  a  feedback  form  that  obtained  their  reactions 
to  the  process. 

Results  showed  that  the  MGEEM  led  to  a  usable  number  of 
indicators  for  each  organization,  with  a  larger  percentage  of 
effectiveness  \ 80  to  95  percent)  as  opposed  to  efficiency 
indicators.  The  process  and  its  results  were  judged  as  very 
acceptable  arid  useful  to  all  but  one  of  the  participants.  The 
indicators  generated  were  judged  cost  effective  to  implement 
since  approximately  80  to  90  percent  made  use  of  existing  data. 
Finally,  the  indicators  were  found  to  have  relatively  low 
consistency  from  organization  to  organization  within  the  three 
functional  areas.  Measures  of  average  pairwise  overlap  for  lists 
of  indicators  within  functional  areas  ranqed  from  11  to 
approximately  19  percent.  Overlap  for  KRAs  was  higher,  ranging 
from  38  to  59  percent. 

Although  the  field  test  demonstrated  that  the  MGEEM  led  to 
results  generally  judged  to  be  useful,  acceptable  and  cost 
effective,  the  lack  of  consistency  in  productivity  indexes  from 
organization  to  organization  indicates  the  need  for  a  follow-up 
process  to  resolve  the  differences  so  as  to  develop  a  set  of 


commonly  applicable  productivity  criteria.  The  paper  offers  some 
additional  modifications  to  the  MGEEM  process  to  improve  the 
consistency  from  site  to  site  and  to  better  "set  the  stage"  for 
participants . 
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I.  INTROMT'l  ION 


Among  the  most  serious  obstacles  to  the  study  of 
productivity  is  the  so-called  "criterion  problem,"  that  is,  the 
measurement  or  assessment  of  productivity  itself.  Most  published 
studies  of  productivity  rely  on  indirect  methods  of  productivity 
measurement,  usually  on  perceptions  of  productivity  reported  by 
supervisors  and  job  incumbents  (e.g.,  Berry  i>  Matthews,  1983; 
Field  &  Hightower,  1983).  Few  investigations  employ  productivity 
criteria  from  engineering  studies  (Tuttle,  1981).  As  a 
consequence  of  the  widespread  use  of  subjective  criterion 
measurement,  the  results  of  many  studies  of  productivity  are  of 
questionable  validity  and  limited  generality.  Efforts  to  enhance 
productivity  in  Air  Force  organizations  have  been  similarly 
hampered  by  a  lack  of  acceptable  productivity  measurement 
procedures . 


Based  on  interviews  with  Air  Force  commanders,  staff 
officers,  and  researchers.  Tuttle  (1981)  developed  the  following 
definition  of  productivity  for  use  in  Air  Force  productivity 
enhancement  efforts; 

Productivity  in  Air  Force  organizations  refers  to  the  volume 
of  resources  used  to  produce  products  and  services 
(efficiency)  and  the  extent  to  which  these  products  and 
services  conform  to  acceptable  standards  of  mission 
performance  (effectiveness)  (Tuttle,  1981,  p.  30). 

Air  Force  commanders  devote  considerable  time  and  resources 
to  the  measurement  and  tracking  of  productivity.  For  example, 
commanders  of  some  organizations  (e.g..  Maintenance)  have  time 
standards  which  permit  the  calculation  of  performance  efficiency 
ratios,  whereas  others  (e.g.,  Weather)  track  after-the-fact 
quality  control  to  assure  the  accuracy  of  their  products  (e.g., 
weather  observations,  forecasts,  and  warnings).  Despite  this 
considerable  investment  in  measurement  data,  productivity 
measurement  practices  vary  considerably  from  organization  to 
organization  in  terms  of  what  is  measured,  how  it  is  measured  and 
how  useful  the  measures  are  (R,  Wilkerson,  personal 
communicati  on,  May  2,  .1982).  As  a  result,  productivity  criteria 
for  cross-organizational  research  are  not  widely  available. 

In  order  to  accurately  assess  organizational  performance, 
evaluate  planning  and  resource  allocation  decisions,  and  conduct 
research  and  development  (R  &  D)  on  the  effectiveness  of 
alternative  management  practices  across  organizational  settings, 
measures  which  include  indicators  of  efficiency  and  effectiveness 
are  required.  To  have  maximum  utility  for  R  &  D  purposes,  the 
measures  should  be  consistent  or  similar  across  organizations 
that  perform  similar  functions,  they  should  cover  all  key  facets 
of  mission  performance,  and  they  should  include  indicators  of 
both  efficiency  and  effectiveness  (Tuttle,  1981).  Moreover, 
indicators  should  possess  the  following  characteristics  (A.T. 


1 


Kearney,  Inc.,  1978,  pp.  11-13): 

1.  Completeness .  All  significant  facets  of  the 
organization's  mission  should  be  covered, 

2.  Comparability .  Measures  should  remain  applicable  from 
one  time  period  to  another, 

3.  Input  coverage .  The  output  indicators  should  cover  all 
significant  results  obtained  from  all  controllable 
inputs  to  the  production  process  (e.g.,  the  results 
produced  by  the  work  hours  of  all  people  working  in  the 
organization ) , 

4.  Compat  ibi  1 ty  with  existing  data  sources  .  Measures 
should  make  maximum  use  of  existing  data  sources, 

5.  Cost-effectiveness .  The  costs  of  measurement  should  not 
exceed  the  benefits  obtained, 

6.  Acceptability .  The  indicators  should  be  meaningful  and 
acceptable  to  those  whose  performance  is  being  measured. 

Finally,  individual  indicators  should  possess  the  following 
characteristics  (Hurst,  1980,  pp.  43-49): 

1.  Validity.  Indicators  should  accurately  reflect  changes 
in  the  organization's  performance, 

2.  Uniqueness .  Indicators  should  be  relatively  independent 
of  each  other, 

3.  Under  s  tanda  b  i  .l.ity.  Individuals  being  measured  should 
understand  how  their  performance  is  reflected  in  the 
indicators  being  used, 

4*  Control  lability .  Organizational  members  should  be  able, 
through  their  actions  on  the  job,  to  produce  changes  in 
che  indicators  which  are  related  to  their  performance, 

5.  Rel iability .  Indicators  should  yield  information  that 
is  repeatable  over  time,  assuming  that  performance 
levels  remain  the  same. 

A  methodology  involving  group  decision-making  procedures  was 
proposed  and  described  by  Tuttle  (1981).  to  generate  objective, 
measures  of  organizational  productivity  in  Air  Force 
organizations.  This  procedure  is  referred  to  as  the  methodology 
for  generating  efficiency  and  effectiveness  measures  (MGEEM)  and 
was  designed  specifically  to  produce  "hard"  measures  of 
organizational  productivity  in  organizations  that  have  few  or  no 
such  existing  measures.  The  purpose  of  the  present  effort  was  to 
conduct  a  field  test  to  evaluate  the  MGEEM  in  terms  of  its 


capability  to  develop  productivity  indicators  for  research  and 
management  purposes.  The  field  test  focused  on  four  questions. 
First,  to  what  extent  does  the  MGEEM  yield  a  usable  number  of 
efficiency  and  effectiveness  measures?  Second,  how  acceptable 
are  the  MGEEM  processes  and  products  to  personnel  within 
organizations  studied?  Third,  how  cost-effective  are  the 
productivity  measures  developed  by  the  MGEEM?  Fourth,  how 
similar  or  consistent  are  measures  generated  by  similar 
organizations? 

With  respect  to  the  fourth  question,  several  a  priori 
predictions  were  made  regarding  the  judged  similarity  between 
organizations.  Factors  which  were  hypothesized  to  contribute  to 
differences  in  KFAs  and  indicators  included  command  differences, 
differences  in  the  extent  to  which  perfcrmance  measurement  is 
institutionalized  within  the  function,  homogeneity  of  the 
organizations,  and  unintentional  differences  produced  by 
inconsistency  of  the  meanurement  coordinators.  For  example, 
those  functions  which  consistently  had  a  single  coordinator 
(Propulsion  and  Weather )  should  he  more  similar  than 
Administration,  which  was  split  between  two  measurement 
coordinators.  Weighing  all  these  factors,  wi thin-organization 
similarity  between  lists  of  KFAs  for  similar  organizations  was 
predicted  to  be  greater  than  the  similarity  between  lists  of 
indicators.  Additionally,  between-organization  similarity  of 
both  indicator  and  KRA  ratings  was  predicted  to  be  greater  for 
Weather  than  for  any  of  the  other  two  functions  and  to  be  greater 
for  Propulsion  than  for  Administration.  The  researchers 
predicted  that  the  highest  similarity  ratings  would  be  obtained 
among  organizations  in  the  Weather  function  because:  all  Weather 
organizations  belong  to  a  single  command;  measurement  of  many 
facets  of  Weather  performance  is  common  practice;  and  although 
personnel  in  Weather  organizations  fall  into  one  of  three  job 
functions,  they  are  highly  interrelated  and  have  a  common  focus. 

The  next  most  similar  indicators  were  predicted  in 
Propulsion.  Although  the  Propulsion  organizations  cut  across 
three  commands,  the  work  is  quite  similar,  performance 
measurement  is  used  extensively  in  maintenance  organizations,  the 
work  performed  is  perhaps  the  most  homogeneous  of  the  three 
functions  studied;  and  a  single  coordinator  was  used.  The  lowest 
similarity  values  were  predicted  for  Administration.  Not  only  do 
the  Administration  organ i zations  span  three  commands,  they  did 
not  employ  measures  of  performance  to  the  extent  that  the  other 
two  functions  did.  In  addition,  the  work  in  an  Administration 
Division  was  separated  into  three  functions  and  these  were 
geographically  separated  as  well.  Finally,  two  measurement 
coordinators  were  employed  in  the  work  with  Administration 
organizations . 
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than  do  indicator  ratings.  This  is  true  for  both 
conunander/deputy  and  research  ratings.  In  general,  the 
similarity  ratings  by  function  conform  to  the  predictions  that 
Administration  had  the  lowest  ratings  for  KRAs  and  indicators 
based  on  ratings  by  both  commanders/deputies 


Table  7.  Average  Percent  of  Overlap  for  KRAs  and  Indicators 


Command/Deputy  Ratings  Researcher  Ratings 


Function 

N  KRAs 

N  Indicators 

N  KRAs 

N  Indicators 

Administration 

37.8 

10.8 

21.6 

6.1 

Propulsion 

58.9 

18.8 

35.1 

11.9 

Weather 

48.6 

18.8 

46.5 

18.9 

and  researchers.  Using  researcher  ratings.  Propulsion  shows  a 
lower  average  similarity  than  does  Weather  for  both  KRAs  and 
indicators.  Since  participant  ratings  for  Propulsion  are  based 
on  incomplete  data  due  to  the  low  response  rate,  the  participant 
ratings  may  be  questionable  and  less  reliable  than  researcher 
ratings . 


17 


Table  6  presents  the  results  of  the  analysis  of  indicators 
with  respect  to  use  of  existing  data.  For  both  Administration 
and  Propulsion,  the  percentages  of  indicators  that  required  no 
new  data  collection  exceeded  90  percent.  For  Weather,  the  figure 
was  79.7. 


Table  6.  Number  and  Percentage  of  Indicators 


Able 

to  be  Formed  from 

Existing  Data  by 

Function 

No.  From 

Percent  From 

Function 

No.  Indicators 

Existing  Data 

Existing  Data 

Administration 

154 

141 

91.6 

Propulsion 

152 

145 

95.4 

Weather 

178 

142 

79.8 

Sima  lari ty  of  KRA 1 s  and  Indicators  Across  Organi zations 

In  t-he  original  statement  of  the  MGEEM  (Tuttle,  1981),  Phase 
IV  of  the  methodology  provided  a  means  of  resolving  differences 
between  results  from  similar  organizations  within  a  function  to 
arrive  at  a  common  set  of  KRAs  and  indicators.  A  set  of 
indicators  which  can  be  applied  across  similar  organizations  is 
necessary  if  they  are  to  be  used  as  productivity  measures  in 
cross-organizational  research.  As  has  been  stated,  one  objective 
of  the  field  test  was  to  determine  the  degree  of  similarity  of 
indicators  generated  from  different  organizations  within  the  same 
function  that  exists,  prior  to  attempts  to  "force"  similarity. 
Therefore,  the  Phase  IV  of  the  MGEEM  design  to  "force"  similarity 
was  not  applied,  in  this  field  test. 

At  least  two  questions  are  involved  in  the  similarity 
analyses.  First,  how  does  one  assess  similarity?  That  is,  what 
is  the  metric?  Second,  given  an  acceptable  metric,  what  is  a 
sufficient  level  of  overlap  <o  justify  the  conclusion  that  the 
MGEEM  produces  a  satisfactory  level  of  similarity? 

The  metric  previously  developed  to  assess  similarity  was  the 
average  percent  of  overlap  between  pairs  of  similar 
organizations .  This  measure  was  computed  separately  for  KRAs  and 
indicators  on  the  basis  of  ratings  made  by  commanders  deputies 
and  researchers. 

Table  7  presents  the  similarity  rating  results.  For  all 
three  functions,  the  KRA  ratings  show  higher  levels  of  similarity 
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Another  comment  from  a  Group  B  participant  illustrates  the 
perceived  educational  effect  the  process  had  on  participants. 

The  following  comment  is  from  an  assistant  crew  chief. 

Helped  me  gain  insight  as  to  the  overall  branch  mission 
and  some  of  the  problems  incurred.  Also,  by  participating 
in  the  process,  I  gained  a  feeling  of  having  my  views 
considered  by  higher  headquarters. 

Two  additional  comments  from  Group  B  participants  illustrate 
the  benefits  of  the  process.  First,  from  a  branch  chief: 

This  type  of  survey  is  greatly  needed  in  our  system. 
Normally,  we  don't  talk  to  each  other  enough. 

Next,  from  a  management  assistant: 

I  feel  we  achieved  a  better  understanding  of  what  our 
problems  are  and  how  to  best  solve  them.  If  we  achieve 
10%  of  what  was  discussed  in  the  sessions,  I  feel  we  were 
successful  in  making  a  change  for  the  better  in  our 
attitudes  and  productivity. 

In  summary,  with  only  one  exception,  the  MGEEM  was  viewed 
favorably  by  unit  commanders.  Other  management  and  non- 
management  participants,  as  a  whole, f elt  that  the  process  and  its 
results  were  quite  acceptable.  Thus,  in  terms  of  participant 
reaction,  the  MGEEM  was  favorably  evaluated. 

Cost  Effectiveness  of  the  Indicators 

A  third  important  aspect  cf  the  field  test  was  determining 
the  extent  to  which  data  to  form  indicators  developed  by  MGEEM 
are  currently  available.  There  are  at  least  three  forms  of 
existing  data.  The  most  obvious  are  numbers  or  entries  in 
particular  blocks  on  existing  reporting  forms.  Another  type  of 
existing  data  includes  entries  on  management  information  system 
products  provided  to  managers/commanders  by  staff  support 
agencies  or  higher  headquarters.  Finally,  there  is  a  variety  of 
local  data  including  status  boards,  customer  feedback  forms 
received,  duty  rosters,  etc.  In  the  latter  case,  data  are  often 
available  but  are  not  tabulated  in  the  format  required  to  form 
the  indicator.  Nevertheless,  all  three  categories  are  considered 
together  in  this  discussion  under  the  heading  of  "existing  data." 
If  an  indicator  required  the  establishment  of  a  new  log  or  some 
other  form  of  initial  data  collection,  the  indicator  was 
considered  not  to  make  use  of  existing  data.  The  fact  that  an 
indicator  for  which  data  are  not  now  available  is  included  in  the 
final  list  means  that  a  unit  commander  has  made  the 
determination  that  the  cost  of  additional  data  collection  is 
warranted  by  the  value  of  the  indicator.  Otherwise  the  indicator 
would  have  been  eliminated  in  the  review  process. 


-  •->  U 
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using  his  suggestions  and  recommendations  in  developing 
key  indicators.  Overall  an  excellent  session. 

Some  commanders  expressed  skepticism  with  the  assumptions 
underlying  the  process,  as  indicated  by  the  following  comment. 

The  approach  was  sound  and  refreshing.  I'm  still  not 
totally  convinced  of  the  validity  of  letting  employees 
develop  indicators  of  our  Key  Result  Areas.  Management 
thinking  is  often  quite  different  from  that  of  the 
employees  when  it  comes  to  measuring  results. 

Several  Group  A  participants  commented  that  the  results  of 
the  process  did  not  surprise  them  or  generate  anything  new.  An 
example  of  such  a  comment  is  the  following: 

Every  KRA  listed  as  important  to  us  merely  repeats  the 
good  management  techniques  in  Air  Force  Regulations  or  Air 
Force  Manuals.  We  proved  that  the  'book*  is  right. 

Only  one  participant  expressed  a  negative  view  toward  the 
process.  This  unit  commander  missed  the  part  of  the  session 
which  described  the  purpose  of  the  process,  but  he  participated 
in  the  remainder  of  a  Group  A  session.  He  stated: 

The  coordinator  did  not  understand  the  structure,  mission, 
or  policy-making  procedures  of  the  Air  Force.  The  overall 
objective  of  a  wing  is  to  mobilize  and  fight  in  a 
deployment  posture.  Air  Force  units  can't  be  measured  the 
same  way  as  a  civilian  company. 

Participants  in  Group  B  had  some  mixed  feelings  about  the 
process.  On  one  hand,  they  found  the  sessions  interesting  and 
different  from  the  usual  work-related  meeting.  On  the  other 
hand,  they  were  skeptical  that  any  one  would  listen  to  their 
ideas  or  that  anything  would  change  as  a  result  of  their 
participation.  These  mixed  attitudes  are  vividly  expressed  by  a 
subordinate  of  the  commander  whose  negative  view  is  expressed 
above. 

I  feel  that  managers  in  Group  B  should  have  had  more  time. 
Perhaps  the  coordinator  should  have  explained  in  more 
detail  how  KRAs  were  developed  in  Group  A.  I  am 
enthusiastic  about  the  procedures  being  used  in  the 
measurement  process  as  they  seemed  to  open  our  eyes  to  the 
other  duties,  requirements,  needs  of  each  DA  section.  You 
had  two  strikes  against  you  from  the  start  since  most 
managers  seem  to  feel  it  was  a  waste  of  time.  The 
management/supervisor/technician  exchanging  ideas  in  this 
type  of  atmosphere  seemed  to  be  very  healthy  for  all  of 
us.  Don't  think  any  of  us  are  certain  that  the  results 
achieved  will  be  applied.  But  I  can  say  that  we  all 
enjoyed  it. 
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The  results  for  Group  A  showed  a  high  degree  of 
understanding  of  the  purpose  of  the  process;  on  a  nine-point 
scale,  the  average  ratings  were  8.3,  8.1,  and  8.3,  respectively, 
for  Administration,  Propulsion,  and  Weather.  The  results  also 
showed  a  very  favorable  reaction  to  the  coordinator  (8.4;  8.2; 
8.8)  and  the  working  climate  created  (8.5;  8.6;  8.7).  The 
consistency  of  these  results  across  functional  areas  suggests 
that  the  work  as  measurement  coordinators  by  the  two  researchers 
was  perceived  similarly  by  participants  in  the  three  functions. 

Using  a  nine-point  scale.  Group  A  participants  in  the  three 
functions  rated  the  MGEEM  task  as  only  moderately  difficult  (5.4; 
4.7;  5.6)  but  interesting  (8.1;  7.8;  8.1).  The  KRAs  were  viewed 
as  acceptable  to  Group  A  members  (8.0;  7.8;  7.9),  as  was  the 
priority  ranking  of  KRAs  (7.9;  7.8;  7.9).  Group  A  rated  itself 
as  very  successful  (8.1;  8.1;  8.1J  and  rated  the  success  of  the 
total  MGEEM  as  only  slightly  less  successful  (7.9;  7.7;  7.4). 

For  all  three  functions,  the  members  of  Group  A  expressed  a 
slight  increase  in  productivity  awareness  as  a  result  of 
participating  in  the  process  (3. 7/4. 3;  4.G/4.9;  3. 8/4.2). 

The  results  from  Group  B  are  very  similar  to  those  for  Group 
A.  Group  B  members  expressed  satisfaction  with  their  success  in 
generating  indicators  (7.5;  8.2;  7.6)  and  felt  that  the  process 
was  beneficial  to  them  in  helping  to  better  understand  their 
organization’s  mission  (7.2;  7.7;  7.3).  As  with  Group  A,  Group  B 
members  expressed  satisfaction  with  the  role  of  the  coordinator 
(7.9;  8.2;  8.5),  the  working  climate  created  (8.4;  8.6;  8.5),  and 
with  the  process  used  (7.9;  8.1?  7.7).  They,  too,  found  their 
task  only  moderately  difficult  (6.3;  5.2;  6.5)  but  interesting 
(7.7;  8.0;  7.4).  Compared  to  Group  A,  members  of  Group  B 
expressed  a  slightly  lower  initial  level  of  productivity 
awareness,  but  also  felt  that  the  process  raised  that  level  of 
awareness  (3. 4/3. 8;  3. 6/4. 2;  3. 2/3.9). 

Further  insight  into  the  meaning  of  these  ratings  can  be 
gleaned  from  the  "write-in"  remarks  of  participants.  The 
following  comments  provide  an  overview  of  the  range  of 
participant  attitudes.  The  commander  of  one  weather  detachment 
observed : 

The  process  was  extremely  beneficial  to  the  people  whole 
unit,  but  especially  Group  B.  They  are  more  aware  of 
management ' s  position  and  feel  they  are  now  participants.  I 
expect  morale  to  improve  ever  if  no  changes  are  made.  This 
plus  the  feedback  are  key  benefits. 

Another  commander  expressed  some  ideas  for  expanding  the  process; 

A  third  session  including  the  commander  with  Group  B 
participants  would  have  been  beneficial  (with  the 
researcher  present).  We  will  eventually  follow-up  with 
this.  Also,  an  extended  session  with  the  coordinator 


Table  5.  Participant  Feedback  Report  Results  -  Group  B 


Administration  Propulsion  Weather 
N  -  54  N  =  59  N  =  30 


Item  Description 

Mean 

SD 

Mean 

SD 

Mean 

SD 

4.  Understanding  the  process 

7.5 

1.8 

7.9 

1.3 

7.7 

1.5 

5.  Understanding  the  instructions 

7.9 

1.5 

8.3 

1.1 

8.2 

0.9 

6.  Importance  of  coordinator 

7.9 

1.6 

8.2 

1.2 

8.5 

0.8 

7.  Coordinator's  attitude 

8.4 

1.1 

8.7 

0.6 

8.9 

0.4 

8.  Working  climate  created 

8.4 

1.2 

8.6 

0.9 

8.5 

0.7 

9.  Time  for  Group  B's  task 

7.6 

1.8 

7.9 

1.7 

7.9 

1.6 

10.  Difficulty  of  Group  B's  task 

6.3 

2.5 

5.2 

2.7 

6.5 

1.8 

11.  Interest  in  Group  B's  task 

7.7 

1.7 

8.0 

1.1 

7.4 

1.6 

12.  Understanding  meaning  of 
Indicator  types 

7.7 

1.5 

7.9 

1.3 

7.6 

1.3 

13.  Round-robin  process 

7.9 

1.5 

8.1 

1.2 

7.7 

1.3 

14.  Acceptability  of  indicators 

7.4 

1.6 

8.1 

1.2 

7.4 

1.1 

15.  Acceptability  of  indicator 
ranking 

7.3 

1.6 

7.9 

1.2 

7.2 

1.4 

16.  Success  of  Group  B 

7.5 

1.5 

8.2 

1.6 

7.6 

1.4 

17.  Benefits  to  you 

7.2 

1.9 

7.7 

1.5 

7.3 

1.5 

18.  Productivity  awareness*3 

Pre  MGEEM  Sessions 

3.4 

0.8 

3.6 

0.9 

3.2 

0.9 

19.  Productivity  awareness 

3.8 

0.9 

4.2 

0.7 

3.9 

0.6 

Post  MGEEM  Sessions 


f*Items  4-17  used  a  nine-point  scale:  l=low,  9=high. 
bItems  18-19  used  a  five-point  scale:  l=low,  5=high. 


Group  A  members  who  were  also  members  of  Group  B  filled  out  a 
Participant  Feedback  Report  for  both  groups. 


Table  4.  Participant  Feedback  Report  Results  -  Group  A 


Administration 

Propulsion 

Weather 

N  =  34 

N  =  29 

N  =  18 

Item  Description 

Mean 

SD 

Mean 

SD 

Mean 

SD 

4.  Understanding  the  process3 

8.3 

1.3 

8.1 

1.3 

8.3 

0.8 

5.  Understanding  the  instructions 

8.5 

1.2 

8.2 

1.3 

8.7 

0.6 

6.  Importance  of  coordinator 

8.4 

1.3 

8.2 

1.3 

8.8 

0.7 

7.  Coordinator's  attitude 

8.6 

1.1 

8.7 

0.9 

8.9 

0.2 

8.  Working  climate  created 

8.5 

1.2 

8.6 

0.9 

8.7 

0.6 

9.  Time  for  Group  A's  task 

8.4 

1.2 

7.9 

1.4 

7.8 

1.8 

10.  Difficulty  of  Group  A's  task 

5.4 

3.0 

4.7 

2.4 

5.6 

2.5 

11.  Interest  in  Group  A's  task 

8.1 

1.4 

7.8 

1.4 

8.1 

1.2 

12.  Understanding  meaning  of  KRAs 

8.3 

1.3 

8.2 

1.1 

8.3 

0.7 

13.  Round-robin  process 

8.3 

1.3 

8.1 

1.1 

8.1 

1.0 

14.  Acceptability  of  KRAs 

8.0 

1.3 

7.8 

1.2 

7.9 

0.9 

15.  Acceptability  of  KRA  ranking 

7.9 

1.2 

7.8 

1.0 

7.9 

0.9 

16.  Success  of  Group  A 

8.1 

1.4 

8.1 

1.2 

8.1 

1.0 

17.  Success  of  total  process-A  &  B 

7.9 

1.3 

7.7 

1.5 

7.4 

1.1 

U. 

18.  Pre-Productivity  awareness 

3.7 

0.9 

4.0 

0.8 

3.8 

0.9 

19.  Past-Productivity  awareness 

4.3 

0.9 

4.9 

0.8 

4.2 

0.6 

f*Items  4-17  used  a  nine-point  scale:  l  =  low,  9=high. 
DItems  18-19  used  a  five-point  scale:  l=low,  5=high. 
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Table  3.  Classification  of  Indicators  by  Function  and  Type 


Indicator 

Type 

Functional 

Area 

Total 

No.  Ind. 

Efficiency 

Effectiveness 

1 

2 

3 

Total 

% 

1 

2 

3 

4 

Total 

% 

Administrati 

on  154 

7 

5 

9 

21 

14 

70 

29 

21 

13 

133 

86 

Propulsion 

152 

4 

z 

2 

8 

5 

74 

31 

28 

11 

144 

95 

Weather 

178 

3 

2 

6 

11 

6 

48 

76 

27 

16 

167 

94 

indicators  were  of  the  effectiveness  type,  while  for 
Administration  units,  86%  were  effectiveness  indicators.  This 
preponderance  of  effectiveness  indicators,  as  compared  to 
efficiency  indicators,  has  several  implications  which  will  be 
addressed  later  in  this  paper. 

Acceptability  of  the  Process  to  Participants 

The  Participant  Feedback  Report  form,  described  in  the 
Methods  Section,  was  used  to  obtain  the  formal  reactions  of 
participants  to  the  Group  A  and  B  sessions.  Results  are 
summarized  in  Table  4  for  Group  A  members  and  Table  5  for  Group  B 
members . 
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Table  2.  Average  Number  of  KRAs  and  Indicators  per  Organization 


No.  KRAs 

No.  Indicators 

No.  Indicators  per  KRA 

Administration 

5.5 

19.3 

3.5 

Propulsion 

4.9 

19.0 

3.8 

Weather 

8.3 

22.1 

2.7 

The  mix  of  efficiency  and  effectiveness  indicators  developed 
by  the  MGEEM  is  shown  in  Table  3.  Indicators  were  classified 
into  two  categories  according  to  the  following  definitions: 

Efficiency  Indicators: 

Type  1  =  Output/Input  (e.g.  No.  bags  of  mail  sorted/No. 
hours  worked) 

Type  2  =  Standard  Output/Actual  Output  (Std.  no.  hours  to 
repair  engine/Actual  no.  hours  required) 

Type  3  =  Actual  Utilization/Potential  Utilization  (No. 

square  feet  warehouse  space  used/No.  square 
feet  available) 

Effectiveness  Indicators: 

Type  1  =  Goals  Achieved  (No.  space  engines 

available/Planned  No.  engines  available) 

Type  2  *  Quality  (No.  weather  observations  submitted 

error  free/No.  weather  observations  submitted) 

Type  3  =  Internal  Impact  (Avg.  No.  work  hours  lost  due  to 
accidents ) 

Type  4  =  External  Impact  (Property  damage  due  to 
unforecast  weather) 

In  each  function  studied,  the  majority  of  indicators  generated 
were  effectiveness  indicators.  The  largest  proportion  of 
effectiveness  indicators  was  found  in  Propulsion  organizations, 
where  95%  of  the  generated  indicators  were  concerned  with 
effectiveness.  For  Weather  organizations,  94%  of  the  generated 
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III.  RESULTS 


Four  questions  were  addressed  in  this  field  test.  First,  to 
what  extent  did  the  MGEEM  produce  a  usable  number  of  both 
efficiency  and  effectiveness  indicators?  Second,  how  acceptable 
were  the  MGEEM  process  and  its  results  to  personnel  within  the 
organizations  visited?  Third,  to  what  extent  did  the  generated 
indicators  make  use  of  existing  data?  Finally,  how  similar  were 
the  generated  KRAs  and  indicators  among  organizations  within 
each  function? 

Selected  KRAs  and  indicators  which  resulted  from  application 
of  the  MGEEM  are  presented  in  Appendices  C  (Administration),  D 
(Propulsion),  and  E  (Weather).  The  indicators  presented  are 
those  which  survived  the  final  review  process  by  commanders  of 
the  target  organizations.  A  similar  list  for  each  of  the  24 
target  organizations,  along  with  the  similarity  ratings,  and 
participant  reactions  to  the  process  comprise  the  raw  data  used 
to  assess  the  utility  of  the  MGEEM. 

Number  and  Types  of  Indicators  Generated 

In  the  application  of  the  MGEEM,  productivity  is  defined  in 
terms  of  efficiency  and  effectiveness.  In  addition,  a  design 
criterion  for  the  methodology  states  that  it  should  produce 
multiple  indicators  which  adequately  cover  the  mission  of  the 
organization.  Thus,  a  basic  concern  of  the  field  test  had  to  do 
with  the  number  and  type  of  indicators  generated.  Does  the 
methodology  yield  an  adequate  number  to  cover  the  major  mission 
facets,  without  producing  so  many  indicators  as  to  make  them 
unmanageable?  Secondly,  does  the  methodology  lead  to  a 
reasonable  mix  of  both  efficiency  and  effectiveness  indicators? 

The  first  of  these  questions  is  addressed  by  the  data  in 
Table  2.  The  average  number  of  KRAs  per  organization  ranged 
from  4.9  for  Propulsion  to  8.3  for  Weather.  This  number  reflects 
differences  in  facilitator  style  as  much  as  it  did  differences  in 
the  organizations.  For  example  in  the  NGT  process,  one 
facilitator  tended  to  force  groups  to  "pick  the  top  5"  KRAs, 
where  as  the  other  researcher  "allowed"  the  groups  to  select  the 
top  7-10.  On  the  average,  the  number  of  indicators  generated  per 
organization  ranges  from  about  19  in  Propulsion  and 
Administration  to  22  in  Weather.  Then  the  number  of  indicators 
did  not  vary  as  much  as  the  number  of  KRAs.  This  is  due  to  the 
fact  that  the  more  KRAs  which  were  generated,  the  fewer  the 
indicators  per  KRA.  The  average  number  of  indicators  per  KRA 
ranges  from  2.7  in  Weather  to  3.8  in  Propulsion. 


commander's  organization  with  the  KRAs  and  indicators  generated 
by  each  of  the  seven  other  organizations  within  the  function. 

The  rater  was  instructed  to  examine  the  items  on  List  1  (the 
rater's  list  of  KRAs),  one  item  at  a  time,  and  determine  if  that 
item  was  the  same  as  or  was  a  subset  of  an  item  on  List  2.  Then, 
similar  comparisons  were  made  between  a  second  pair  of 
organizations  until  all  comparisons  were  made  between  the  rater's 
organization  and  the  seven  others,  separately  for  KRAs  and 
indicators.  Thus,  each  commander  rated  the  similarity  between 
all  possible  pairings  of  the  rater's  organization's  KRAs  and 
indicators  with  the  other  seven  in  the  same  function. 

To  provide  a  second  set  of  similarity  ratings,  the 
researchers  also  rated  KRAs  and  indicators  for  all  possible 
pairings  of  organizations  within  each  of  the  three  functions. 
Researcher  ratings,  presumably  using  a  consistent  frame  of 
reference  across  all  organizations,  provided  a  "second  opinion" 
on  the  similarity  of  KRAs  and  indicators  across  like 
organizations . 

Similarity  scores  for  pairs  of  organizations  were  computed 
by  dividing  the  number  of  items  indicated  as  being  similar  on 
both  lists  by  the  total  number  of  items  on  the  two  lists  and  then 
multiplying  by  100.  If  all  items  on  the  two  lists  being  compared 
were  rated  as  similar,  the  similarity  (or  percentage  of  overlap) 
would  be  100%.  If  no  items  were  rated  as  similar  on  either  list, 
indicating  that  the  two  lists  had  no  items  in  common,  the 
percentage  of  overlap  would  be  0%.  Similarity  scores  were 
computed  for  each  pair  of  organizations  within  each  function, 
based  both  on  participants'  (i.e.,  commanders  and  deputies)  and 
researchers'  ratings. 

Data  from  commanders/deputies  were  obtained  from  rating 
booklets,  which  were  mailed  several  months  after  the  field 
visits.  For  Administration,  the  return  rate  was  50  percent  for 
KRA  ratings  and  25  percent  for  indicator  ratings.  In  Propulsion, 
the  return  rates  were  25  percent  for  KRAs  and  19  percent  for 
indicators.  In  Weather,  the  return  rates  were  88  percent  for 
KRAs  and  69  percent  for  indicators.  Thus,  the  similarity 
analyses  are  based  on  a  less-than-desirable  number  of  ratings, 
especially  in  the  Propulsion  function. 


usually  required  to  achieve  consensus. 

Following  the  generation  of  KRAs  by  Group  A,  Group  B  was 
formed.  Group  B  consisted  of  all  members  of  Group  A  (except  the 
commander)  and  their  immediate  subordinates.  Group  B  was  tasked 
to  develop  and  prioritize  "indicators"  of  efficiency  and 
effectiveness  for  each  KRA.  Using  the  NGT,  a  list  of  indicators 
was  developed  for  each  KRA. 

Each  organization  in  the  study  was  visited  by  a  researcher 
for  5  days.  On  the  first  day,  an  inbriefing  and  familiarization 
with  the  subject  organization  was  conducted.  On  the  second  day. 
Group  A  was  formed  and  KRAs  were  developed.  Days  three  and  four 
involved  the  generation  of  indicators  by  Group  B.  Day  five 
consisted  of  a  review  of  the  KRAs  and  indicators  with  the 
commander  of  the  organization.  In  addition,  this  discussion  with 
the  commander  identified  existing  data  sources  which  could 
provide  information  required  to  form  the  indicators  in  actual 
operational  use. 

Tuttle  (1981)  provides  a  more  extensive  treatment  of  the 
rationale  underlying  the  MGEEM  and  its  development. 

Participant  Feedback 

Following  the  MGEEM  implementation,  The  Participant  Feedback 
Report  was  used  to  obtain  participant  reactions.  The  Participant 
Feedback  Report  consists  of  20  questions.  Examples  of  the 
feedback  forms  sent  to  members  of  Teams  A  and  B  are  found  in 
Appendices  A  and  B  respectively.  Items  1-3  were  demographic 
questions,  and  item  20  was  an  open-ended  item.  The  remaining 
items  used  a  Likert-type  scale  to  rate  various  aspects  of  the 
MGEEM  and  its  results.  The  anchor  terms  varied  somewhat  from 
item  to  item,  but  the  least  favorable  reaction  was  usually 
weighted  a  "1"  and  the  most  favorable  reaction  a  "9"  (items  4-17) 
or  a  "5"  (items  18-19). 

MGEEM  Follow-Up 

Following  completion  of  the  base  visits,  a  list  of  final 
KRAs  and  indicators  developed  by  each  organization  was  prepared. 
These  lists  were  mailed  to  unit  commanders  for  review  and 
comment.  After  final  changes  suggested  by  commanders  were  made 
in  the  lists,  rating  booklets  were  prepared  for  the  final  phase 
of  data  collection.  This  involved  assessment  of  the  similarity 
of  KRAs  and  indicators  across  organizations  within  each 
functional  area. 

Similarity  Assessment 

The  assessment  of  similarity  was  accomplished  through 
ratings  by  unit  commanders  or  their  deputies.  The  rating  task 
involved  comparing  lists  of  KRAs  and  indicators  obtained  from  the 
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Command 


Base 


Weather  Administration  Propulsion 


ATC* 

Williams  AFB,  AZ 

X 

MACb 

Laughlin  AFB,  TX 

X 

MAC 

Andrews  AFB,  MD 

X 

X 

X 

MAC 

McChord  AFB,  WA 

X 

X 

X 

MAC 

Travis  AFB,  CA 

X 

X 

X 

SACC 

Barksdale  AFB,  LA 

X 

SAC 

Griffiss  AFB,  NY 

X 

X 

X 

SAC 

March  AFB,  CA 

X 

X 

X 

TAC° 

Bergstrom  AFB,  TX 

X 

X 

TAC 

Luke  AFB,  AZ 

X 

X 

TAC 

Shaw  AFB,  SC 

X 

X 

f*Air  Training  Command  (ATC) 
Military  Airlift  Command  (MAC) 
^Strategic  Air  Command  (SAC) 
^Tactical  Air  Command  (TAC) 


MGF.EM  Methodology 

The  MGEEM  involves  a  group  decision-making  process  known  as 
the  Nominal  Group  Technique  (Delbecq,  Van  de  Ven,  and  Gustafson, 
1975).  The  Nominal  Group  Technique  (NGT)  consists  of  six  steps: 
(a)  silent  generation  of  ideas  by  individual  group  members,  (b) 
round-robin  listing  of  ideas,  (c)  discussion  and  clarification  of 
ideas  developed,  (d)  voting  to  prioritize  items  from  the  list, 

(e)  further  discussion  and  clarification  of  items  and  voting 
patterns,  and  (f)  additional  voting  and  discussion,  if  necessary, 
to  achieve  consensus.  The  NGT  requires  a  skilled  group 
facilitator  or  measurement  coordinator  to  conduct  the  process. 

The  facilitator,  while  guiding  the  group  in  making  decisions, 
must  not  attempt  to  lead  the  group  toward  any  particular 
decision.  In  the  field  test,  the  researchers  performed  the  role 
of  facilitator. 

The  NGT  process  was  used  in  the  field  test  to  generate:  (a) 
Key  Result  Areas  (KRAs)  and  (b)  Indicators.  Two  groups  of 
organizational  members  were  involved  in  the  NGT  process.  The  KRA 
development  group,  Group  A,  consisted  of  the  organization's 
commander  and  representatives  from  the  next  lower  level  of 
management.  KRAs  were  generated  in  response  to  the  question, 
"What  results  does  the  Air  Force  pay  this  organization  to 
accomplish?"  Potential  KRAs  were  proposed  by  members  of  Group  A 
which  were  presumed  to  tap  the  basic  facets  of  the  organization's 
mission.  Group  A  then  voted  to  prioritize  six  to  nine  KRAs.  Two 
or  three  rounds  of  voting,  each  followed  by  discussion,  were 


II.  METHOD 


Function  and  Base  Selection 

The  function  selection  process  was  guided  by  five  criteria. 
First,  in  order  for  the  results  of  the  field  test  to  be 
generalizable,  the  functions  selected  should  be  qualitatively 
different  from  each  other.  Second,  the  functions  should  be 
important  to  the  Air  Force  in  terms  of  the  number  of  people 
involved  and  in  terms  of  the  relationship  of  the  function  to  the 
primary  flying  mission  of  the  Air  Force.  Third,  the  functions 
selected  should  be  ones  for  which  measures  do  not  currently  exist 
in  the  Air  Force  productivity  measurement  program,  but  which  are 
under  consideration  for  inclusion  in  this  program.  Fourth,  there 
should  be  a  sufficient  number  of  people  at  a  base  in  the  function 
to  permit  formation  of  the  MGEEM  teams  without  excessively 
disrupting  the  organization's  normal  work  performance.  Finally, 
the  functional  area  should  not  be  so  technical  that  the 
productivity  criteria  developed  would  only  have  meaning  for 
specialists  in  the  functional  area;  i.e.,  the  technology  involved 
in  the  function  should  be  comprehensible  to  a  lay  person. 

Consultation  with  representatives  of  the  Air  Force 
Productivity  and  Research  Office  ( AF/MPME )  and  the  project 
monitor  at  the  Air  Force  Human  Resources  Laboratory  ( AFHRL/MODP ) 
resulted  in  the  selection  of  three  functions.  Their  Air  Force 
Functional  Account  Codes  were  Administration  (11XXXX),  Aircraft 
Maintenance — Propulsion  (232XXX),  and  Weather  ( 34XXXX ) .  Within 
these  functions,  the  organizational  units  selected  were  the 
Central  Base  Administration  Division,  Aircraft  Maintenance — 
Propulsion  Branch,  and  the  Base  Weather  Detachment.  For  economic 
reasons,  a  decision  was  made  to  restrict  the  field  test  to  24 
organizations:  eight  in  each  of  the  three  functions.  Thus,  the 
base  selection  problem  concerned  how  to  select  the  minimum  number 
of  bases  in  order  to  satisfy  the  following  sampling  criteria. 
First,  there  should  be  a  minimum  of  20  people  in  each  target 
function  (i.e..  Administration,  Propulsion,  or  Weather)  at  the 
base.  Second,  the  bases  should  represent  at  least  three  Major 
Commands  (MAJCOMS).  Third,  ideally,  more  than  one  function  would 
be  located  at  a  selected  base,  to  minimize  travel  time  and  costs. 
Fourth,  the  selected  bases  should  be  "typical"  of  the  Air  Force 
(e.g.,  not  a  command  headquarters). 

Application  of  the  criteria  led  to  the  selection  of  the 
bases  shown  in  Table  1.  A  total  of  11  bases  were  selected,  which 
provided  the  24  target  organizations.  Four  MAJCOMS  were 
represented,  and  all  but  three  of  the  bases  had  at  least  two 
target  organizations. 


IV.  DISCUSSION 


The  purpose  of  this  investigation  was  to  conduct  a  field 
test  of  the  MGEEM  in  order  to  determine  its  applicability  for 
generating  appropriate  research  criteria  and  use  as  a  management 
tool.  This  section  will  consider  the  evidence  in  these  two  areas 
and  will  also  consider  implications  for  additional  research  and 
suggested  modifications  to  the  MGEEM. 

Implications  for  Use  of  MGEEM  in  Inter-Organizational  Research 

Use  of  the  MGEEM  for  research  purposes  requires  that  the 
process  should  lead  to  a  set  of  reliable,  valid  and  important 
indicators  which  are  common  to  a  range  of  similar  organizations 
and  that  the  process  is  acceptable  to  unit  commanders  and 
organization  members.  This  study  yielded  KRAs  with  moderate 
similarity  between  organizations  and  indicators  with  generally 
low  similarity.  While  the  original  statement  of  MGEEM  (Tuttle, 
1981 )  included  a  phase  designed  to  produce  a  common  set  of 
indicators,  this  phase  was  intentionally  omitted  in  the  present 
research  to  determine  similarity  levels  when  KRAs  and  indicators 
are  free  to  vary.  Therefore,  this  research  demonstrated  that  a 
commonizing  phase  is  necessary  in  order  to  ontain  sufficiently 
high  levels  of  similarity  to  permit  cross-organizational 
research.  however,  the  process  did  yield  indicators  which  were 
judged  important  and  acceptable  to  unit  commanders.  In  order  to 
assess  reliability  and  validity,  other  chan  face  validity,  the 
indicators  must  be  made  operational  and  assessed  statistically. 

Assessments  of  commonality,  when  each  organization  was 
completely  free  to  generate  its  own  KRAs  and  indicator  set, 
demonstrate  that  similar  organizations  develop  Key  Result  Areas 
and  indicators  which  show  only  low  to  moderate  inter-organization 
similarity.  This  provides  evidence  that  the  commonizing  phase  of 
MGEEM  j.s  necessary  in  order  to  obtain  a  common  set  of  indicators 
for  use  across  similar  organizations. 

To  understand  the  meaning  of  the  similarity  scores,  refer  to 
Tables  8  and  9.  Table  8  lists  KRAs  for  uwo  Weather 
organizations.  Organization  ?  has  seven  KRAs  and  organization  5 
has  i.O  KRAs.  These  two  organizations  received  a  KEA  similarity 
value  of  70%.  This  computation  procedure  is  presented  in  Table 


Table  8:  Comparison  of  KRAs  for  Weather  Organizations  2  and  5 


Weather  Organization  2  Weather  Organization  5 


KRA  1-Training  and  Maintain  KRA  1-Issue  accurate  terminal 

Skills  required  for  wartime  area  forecasts 

mission 

KRA  2-Accurately  and  timely 
KRA  2-Provide  weather  warnings  record  and  transmit  weather 
and  advisories  of  hazardous  observations 

weather 

KRA  3-Issue  timely  and  accurate 
KRA  3-Provide  accurate  weather  warnings 

forecasts  for  use  by  local 

commanders  KRA  4-Issue  metwatch  forecast 

advisories 

KRA  4-Provide  current  weather 

observations  locally  to  KRA  5-Prepare  SAC  alert 

military  and  civilian  users  and  packages 
transmit  them  to  national  and 

international  users  KRA  6-Provide  routine  and 

special  briefings  to  aircrews 

KRA  5-Carry  out  management  and 

supervisory  duties  required  to  KRA  7-Meet  the  needs  of 
maintain  skilled  and  motivated  operational  requirements  and 
personnel,  serviceable  unit  personnel  through  good 

equipment,  necessary  records,  personnel  management 

ar.d  adequately  kept  facilities 

KRA  8-Provide  staff  weather 

KRA  6-Br:.ef  outbound  crews  on  support  to  Decision-makers  in 

in-route  and  landing  weather  completing  their  missions 

world  wide  and  metwatch 

aircraft  as  required  KRA  9-Conduct  job  related 

training 

KRA  7-Provide  social  forecasts 

as  required  KRA  10-Provide  capability  to 

support  EWO/contingencies 
operations 


KRAs  1,  3,  4  and  6  from  Organization  2  were  substantially 
the  same  as  KRAs  10,  1,  2  and  6  in  Organization  5.  KRA  2  (Org. 
2)  was  more  general  than  any  KRA  in  Org.  5  in  that  it  encompased 
KRA  No.  3  and  4  (Org.  5).  Therefore  KRA  2  (Org.  2)  was  not 
counted  as  being  the  same  as  any  KRA  in  Org.  5.  However,  KRAs  3 
and  4  (Org.  5)  were  counted  as  being  substantially  the  same  as 
KRA  2  in  Org.  2.  Likewise,  KRA  5  (Org.  2)  encompassed  but  was 
not  identical  to  KRAs  7  and  9  (Org.  5).  KRAs  7  (Org.  2)  and  5 
and  8  (Org.  5)  were  unique.  Therefore,  as  shown  in  Table  9  this 


leads  to  12  out  of  17  KRAs  being  considered  substantially  the 
same  between  the  two  organizations,  yielding  a  value  of  70% 
overlap. 


Table  9:  Analysis  of  KRA  Similarity  Determination 


KRA  No.  -  Organization  2  Comparable  KRA(s)  -  Organization  5 


Similarity  Computation  =  No.  KRAs  in  Common  (Org.  2  vs.  Org.  5) 


Total  Combined  No.  KRAs 


=  12 


=  .70 


To  illustrate  how  the  commonizing  process  for  MGEEM  would 
work  to  increase  similarity,  consider  KRA  2  for  Org.  2.  This  KRA 
covers  both  warnings  and  metwatch  advisories.  Organization  5 
chose  to  split  these  into  two  KRAs.  If  the  commonizing  process 
were  used  to  resolve  this  difference  between  these  two 
organizations  in  terms  of  the  level  of  specifity  of  the  item, 
"commonality"  would  be  increased.  Similarily,  this  would  be 
expected  to  occur  for  KRA  5  (Org.  2)  and  KRAs  7  and  9  (Org.  5). 
The  unique  KRAs  7  (Org.  2)  and  5  and  8  (Org.  5)  may  not  be 
resolved.  For  example,  "social  forecasts,"  e.g.  whether  or  rot 
it  will  rain  on  the  general's  cocktail  party,  are  provided  by  all 
weather  detachments  but  this  does  not  constitute  a  significant 


part  of  a  unit's  workload  and,  therefore,  probably  does  not 
belong  in  a  productivity  measurement  system.  On  the  other  hand, 
"preparing  SAC  alert  packages"  would  be  unique  to  weather 
detachments  located  on  SAC  bases;  therefore,  it  is  a  unique  and 
significant  mission  requirement. 

Applying  the  commonizing  phase  of  MGEEM  to  these  two 
organizations  would  likely  raise  overlap  from  70%  to  at  least 
87%.  This  is  because  KRA  7  (Org.  2)  would  drop  out,  and  a  common 
level  of  specificity  would  be  agreed  upon  for  KRAs  2  and  5  (Org. 
2)  and  3,  4,  7,  9  (Org.  5). 

Since  indicators  are  developed  from  KRAs,  any  differences 
in  KRAs  are  magnified  in  the  indicator  development  process. 

Thus,  it  is  not  surprising  that  indicator  similarity  is 
considerably  lower  than  KRA  similarity.  However,  as  illustrated 
above  for  KRAs,  the  commonizing  process  if  applied  to  indicators 
would  be  expected  to  lead  to  a  substantial  increase  in  similarity 
scores  between  similar  organizations.  The  magnitude  of  the 
expected  gain  will  vary  depending  on  the  "true"  similarity  of 
mission  requirements  between  presumably  similar  organizations. 
However,  it  would  seem  highly  likely  that  a  sufficient  level  of 
similarity  could  be  attained  to  permit  development  of  a  core  set 
of  indicators  for  use  in  research  across  similar  organizations. 
Additional  research  with  various  commonizing  strategies  will  be 
required  to  substantiate  this  hypothesis. 

Implications  for  Use  of  the  MGEEM  Within  Organizations 

The  results  of  the  field  test  clearly  demonstrate  that  the 
MCEEM  is  useful  in  generating  productivity  indicators  for  uses 
within  organizations.  These  uses,  both  diagnostic  and 
therapeutic  in  nature,  do  not  appear  to  be  affected  by 
limitations  in  inter-organizational  generality.  The  high 
acceptability  of  the  MGEEM  to  field  test  participants  and  its 
apparent  ability  to  utilize  existing  data  sources  for  most 
indicators  developed  underscore  the  potential  utility  of  the 
methodology  as  a  management  tool. 

Efficiency  vs .  Effectiveness  Indicators  -  What 1 s  Appropriate 

A  very  high  percentage  of  indicators  generated  were 
effectiveness  as  opposed  to  efficiency  indicators.  The  reasons 
for  this  are  not  clear.  The^e  is  some  evidence  from  anecdotal 
data  collected  by  the  researchers  following  the  group  sessions 
that  efficiency  is  not  a  major  concern  of  Group  B  participants. 
Group  B  members  tended  to  look  at  costs  and  resource  allocation 
as  problems  for  higher  management;  thus,  they  did  not  suggest 
many  efficiency  indicators.  These  individuals  tended  to  view 
resources  as  "given."  Their  major  concern  was  with  getting  the 
job  done.  Some  participants  stated  that  if  they  ran  out  of 
resources  (fuel,  material,  supplies,  etc.),  more  would  be 
provided  because  the  job  must  be  done.  Thus,  there  appeared  to 


be  little  incentive  at  this  level  to  minimize  resource 
expenditures.  This  relative  lack  of  concern  for  efficiency 
contrasts  sharply  with  attitudes  expressed  by  base,  squadron, 
and  wing  commanders  who  were  interviewed  during  the  research. 
Commanders  were  quite  concerned  with  efficiency  and 
effectiveness.  With  respect  to  the  Air  Force  productivity 
improvement  efforts,  an  important  organizational  issue  is  what 
should  be  the  level  of  concern  with  efficiency  vs.  effectiveness 
at  various  organizational  levels.'  Once  that  is  answered,  the 
next  question  relates  to  the  organizational  and  managerial 
factors  which  promote  concern  for  efficiency  in  those 
organizations  where  it  is  most  appropriate.  Based  on  the  results 
of  this  study,  there  appear  to  be  differences  between 
organizations  in  the  extent  to  which  efficiency  is  a  priority. 
Possible  factors  affecting  this  concern  seem  to  be  level  of  the 
organization  (division  vs.  branch),  civilian  vs.  military  mix, 
nature  of  the  work  performed,  etc.  This  issue  also  appears 
worthy  of  additional  research.  It  is  potentially  the  most 
significant  research  issue  identified. 

Implications  for  Modifications  to  MGEEM 

A  number  of  observations  from  the  field  test  experience  with 
the  MGEEM  resulted  in  suggestions  for  further  improvement  of  the 
procedure.  For  instance,  it  became  clear  that  the  measurement 
coordinator  must  be  skilled  in  conducting  group  decision-making 
sessions.  The  importance  of  this  factor  is  difficult  to 
overstate.  A  poorly  trained  or  poorly  skilled  measurement 
coordinator  could  lose  control  cf  the  process  and  less  valid 
indicators  might  be  developed.  In  preliminary  sessions  with 
participants,  the  measurement  coordinator  should  provide  explicit 
examples  of  KRAs  and  indicators  and  describe  the  whole  process  to 
participants.  Questions  should  be  posed  carefully  and 
unambiguously.  From  an  organizational  viewpoint,  it  should  be 
made  clear  to  all  participants  that  the  unit  commander  fully 
supports  the  process.  Additionally,  for  use  in  operational 
settings  the  MGEEM  requires  somewhat  more  time  than  W3S  allocated 
during  the  research  program.  While  one  3-hour  session  was 
sufficient  to  develop  KRAs,  two  3-hour  sessions  were  needed  to 
develop  a  comprehensive  list  of  indicators  for  all  KRAs. 

Following  these  structured  sessions,  group  meetings  or  "sanity 
checks"  with  Group  A  and  Group  B  for  the  purpose  of  reviewing  and 
"polishing"  the  KRAs  and  indicators  would  be  recommended. 

Finally,  the  quality  of  the  results  depends  on  the  abilities  of 
the  participants.  Experienced,  knowledgeable,  and  highly  verbal 
team  members  yield  the  best  results.  Accordingly,  efforts  should 
be  made  to  assign  such  personnel  to  participate  in  the  MGEEM 
process . 

These  observations  have  been  incorpoi ated  into  a  modified 
MGEEM,  which  is  outlined  in  Appendix  F. 
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V.  CONCLUSIONS 


The  field  test  of  the  MGEEM  methodology  demonstrated  the 
following.  First,  the  MGEEM  led  to  a  set  of  indicators  judged 
important  and  feasible  by  commanders.  Second,  there  was  moderate 
similarity  between  organizations  within  a  function  with  respect 
to  KRAs  and  generally  low  similarity  between  organizations  with 
respect  to  indicators.  A  seemingly  disproportionate  subset  of 
the  indicators  developed  were  measures  of  effectiveness  as 
opposed  to  efficiency.  Third,  the  process  was  highly  acceptable 
to  participants.  Fourth,  the  indicators  would  appear  to  be  cost 
effective  to  implement  since  a  high  percentage  make  use  of 
existing  data.  These  findings  have  implications  for  both 
research  applications  of  the  methodology  and  for  organizational 
productivity  measurement  and  enhancement  applications. 
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Base 


Target  Work  Center 


Participant  Feedback  Report  --  Team  A 
Instructions 

You  recently  participated  in  a  measurement  process  con¬ 
ducted  by  the  Maryland  Center  for  Productivity  and  Quality  of 
Working  Life  designed  to  generate  Key  Results  Areas  and 
productivity  Indicators  for  the  work  center  indicated  at  the 
top  right  of  this  form.  Please  indicate  your  reactions  to 
these  procedures  and  the  results  by  responding  to  the  follow¬ 
ing  questions.  Thank  you  for  your  cooperation. 

Background  Information 

1.  Your  Position  Title/Function  _ _ _ _ _______ 

2.  Length  of  time  in  your  present  position  _ years  _ months 

3.  (a)  Status _ Military  (b)  TAFMS : _ years  _ months 

Civilian 

Reactions  to  the  Process 


Please  circle  the  number  which  best  indicates  your  reaction 
to  the  element  of  the  measurement  process  listed  below. 

4.  Your  understanding  of  the  purpose  of  the  measurement  process 
Poor  Understanding  123456789  Good  Understanding 

5.  Instructions  provided  by  the  coordinator 

Not  Clear  123456789  Clear 

6.  Importance  of  a  coordinator  to  the  process 

Not  Important  123456789  Important 

7.  The  coordinator's  attitude  toward  the  process 

Negative  123456789  Positive 

8.  The  working  climate  created  by  the  coordinator 

Not  Conducive  to  123456789  Conducive  to 
Open  Discussion  Open  Discussion 
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9.  The  time  allocated  for  Team  A's  task 

Insufficient  123456789  Sufficient 

10.  The  difficulty  of  Team  A's  task  (generating  and  ranking 
Key  Results  Areas) 

Not  Difficult  123456789  Difficult 

11.  Your  reaction  to  Team  A's  task 

Not  Interesting  123456789  Interesting 

12.  Your  understanding  of  the  meaning  of  Key  Results  Areas 

Not  Clear  123456789  Clear 

13.  The  round-robin  process  used  to  generate  Key  Results  Areas 

Not  Successful  123456789  Successful 

14.  The  acceptability  to  the  members  of  your  Team  of  the  Key 
Results  Areas  generated 

Not  Acceptable  123456789  Acceptable 

15.  The  acceptability  to  the  members  of  your  Team  of  the 
priority  ordering  of  Key  Results  Areas 

Not  Acceptable  123456789  Acceptable 

16.  How  would  you  rate  the  overall  success  of  Team  A  in 
developing  a  useful  set  of  Key  Results  Areas 

Not  Successful  123456789  Successful 

17.  How  would  you  rate  the  success  of  the  total  process 
(Teams  A  and  B)  in  developing  a  useful  set  of  Key 
Results  Areas  and  Indicators 

Not  Successful  123456789  Successful 

18.  Prior  to  participating  in  this  measurement  process,  how 
would  you  describe  your  level  of  awareness  of  the  meaning 
of  productivity  in  Air  Force  Orqani zations ? 

(Check  one) 

1.  _  Very  Low 

2 .  _  Low 

3.  _ Moderate 

4 .  _  High 

5.  _  Very  High 
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19.  As  a  result  of  participating  in  this  process,  how  would 
you  describe  your  current  level  of  awareness  of  the  meaning 
of  productivity  in  Air  Force  organizations? 

(Check  one) 

1.  _  Very  Low 

2 .  _  Low 

3.  _  Moderate 

4.  _  High 

5.  _  Very  High 

20.  What  other  reactions  do  you  have  to  the  process,  its 
results,  or  its  utility  for  your  organization? 
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Base 


Target  Organization 


Participant  Feedback  Report  --  Team  B 
Instructions 


You  recently  participated  in  a  measurement  process  con¬ 
ducted  by  the  Maryland  Center  for  Productivity  and  Quality  of 
Working  Life  designed  to  generate  productivity  Indicators  for 
the  organization  listed  in  the  top  right  corner  of  this  form. 
Please  indicate  your  assessment  of  this  process  and  the  re¬ 
sulting  Indicators  by  responding  to  the  following  questions. 

Background  Information 

1.  Your  Position  Title/Function _ _ _ _______ _ _ _ _ _ 


2.  Length  of  time  in  your  present  position  year  _ month 

3.  (a)  Status  _ _  Military  (b)  TAFMS :  _ year  _ month 

Civilian 


React  ions  to  the  Process 

Please  circle  the  number  which  best  indicates  your 
reaction  to  the  element  of  the  measurement  process  listed 
below . 

t .  Your  understanding  of  the  purpose  of  the  measurement  process 
Poor  Understanding  123456769  Good  Understanding 

5.  Instructions  provided  by  the  coordinator 

Not  Clear  123456789  Clear 

6.  Importance  of  the  coordinator  to  the  process 

Not  Important  123456789  Important 

7.  The  coordinator's  attitude  toward  tie  process 

Negative  123456789  Positive 

8.  The  working  climate  created  by  the  coordinator 

Not  Conducive  to  12  3456789  Conducive  to 
Open  Discussion  Open  Discussion 
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9.  The  time  allocated  for  Team  B's  task  (generating  and 
ranking  Indicators  for  Key  Results  Areas) 

Insufficient  123456789  Sufficient 

10.  The  difficulty  of  Team  B's  task 

Not  Difficult  123456789  Difficult 

11.  Your  reaction  to  Team  B's  task 

Not  Interesting  123456789  Interesting 

12.  Your  understanding  of  the  meaning  of  efficiency  and 
effectiveness  Indicators 

Not  Cearl  123456789  Clear 

13.  The  round  robin  process  used  to  generate  Indicators 

Not  Successful  123456789  Successful 

14.  The  acceptability  to  the  members  of  your  Team  of  the 
Indicators  generated 

Not  Acceptable  123456789  Acceptable 

15.  The  acceptability  to  members  of  your  Team  of  the  priority 
ordering  of  Indicators 

Not  Acceptable  123456789  Acceptable 

16.  How  would  you  rate  the  overall  success  of  Team  B  in 
developing  a  useful  set  of  Indicators 

Not  Successful  123456789  Successful 

17.  The  benefits  to  you  of  the  process  in  terms  of  providing 

a  better  understanding  of  the  total  organization ' s  mission 

Not  Beneficial  123456789  Beneficial 

18.  Prior  to  participating  in  this  measurement  process,  how 
would  you  describe  your  level  of  awareness  of  the  meaning 
of  productivity  in  Air  Force  organizations? 

(Check  one) 

1.  _  Very  Low 

2 .  _  Low 

3 .  Moderate 

4.  _  High 

5.  ______  Very  High 
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The  MGEEM 
Phase  1: 
Phase  2 : 
Phase  3 : 
Phase  4: 
Phase  5: 
Phase  6 : 

Phase  1: 
*1.1. 
1.2. 
*1.3. 
*1.4. 

*1.5. 
*1.6. 
Phase  2: 
*2.1. 
*2.2. 

2.3. 

2.4. 
Phase  3: 

3.1. 

3.2. 


Revised  MGEEM  —  Outline  of  Process 

consists  of  six  phases: 

Decision  to  Measure  Productivity 
Organizational  Familiarization 
Development  of  Key  Result  Areas 
Development  of  Indicators 

Refinement  of  Indicators  and  Data  Sources 
Generalization  of  Indicators 

Decision  to  Measure  Productivity 

Defining  goals  of  the  measurement  activity 

Select  measurement  coordinator 

Develop  a  working  definition  of  productivity 

Analyze  forces  for  and  against  implementation  of 
the  measurement  methodology 

Finalize  measurement  plan 

Obtain  necessary  organizational  commitments 
Organizational  Familiarization 
Review  existing  documentation 

Visit  the  target  organization  or  a  similar  organization 

Define  organizational  boundaries 

Construct  organizational  diagram 

Development  of  Key  Result  Areas 

Form  Team  A 

Orient  Team  A 


*An  asterisk  denotes  those  steps  which  have  been  added  or  sig¬ 
nificantly  modified  in  the  restatement  of  the  MGEEM. 
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APPENDIX  F 


7-3.  Average  number  shifts  per  individual  Local  Records, 

shift  worker  per  month  (also  forecaster.  Not  Tabulated 
Wing  Weather  Officer,  etc.) 

7-4.  Number  personnel  nominated  for  awards  Local  Records, 

Not  Tabulated 


KRA  8:  Provide  staff  weather  support  to  assist  decision  makers 
in  completing  their  missions. 

8-1.  Inspector  General  inspection  ratings  I.G.  Report 

8-2.  Operational  Readiness  Inspection  ratings  O.R.I.  Report 


KRA  9:  Conduct  job-related  training. 

9-1.  Number  fully  trained  in  a 

category _  _ 

Number  personnel  to  be  trained 
in  the  category 

9-2.  Number  personnel  upgraded  on 

time _ _ 

Number  personnel  eligible  for 
upgrading 


Unit  Training 

Records _ 

Unit  Training 
Records 

Unit  Records, 

CBPO  Data _ 

Unit  Records, 
CBPO  Data 


KRA  10:  Provide  the  capability  to  support  contingency 
operations . 

10-1.  Number  discrepancies  in  local  Base  Mobility 

mobility  exercise  Exercise  Report 

10-2.  Evaluation  of  mobility  capability  I.G.  Report 

during  I.G.  inspections 


3-4 .  Number  after  the  fact  warnings  issued 
Total  number  warnings  issued 


3rd  W  W  Form  16 
3rd  W  W  Form  16 


3-5.  Number  compliance  errors  on  AWS  Form  39 
Number  possible  errors 


AWS  Form  80 
AWS  Form  80 


KRA  4:  Issue  metwatch  forecast  advisories. 

4-1 .  Number  metwatch  forecasts  verified 
Number  metwatch  forecasts  issued 

4-2.  Number  bust  reviews  for  metwatch 
forecasts 

4-3.  Number  compliance  errors  in  metwatch 

forecast  advisories _ 

Number  possible  errors 


AWS  Form  15 
AWS  Form  1 5 

AWS  Form  15 


AWS  Form  80 
AWS  Form  80 


KRA  5:  Prepare  SAC  alert  packages. 

5-1.  Number  compliance  errors  in  SAC  alert 

packages _ 

Number  possible  errors 


AWS  Form  80 
AWS  Form  8  0 


KRA  6:  Provide  routine  and  special  weather  briefings  to 
aircrews  and  commanders  to  support  operational 
missions . 


6-1.  Number  compliance  errors  on  175-1, 

SAC  597-598  _ AWS  Form  80 

Number  possible  errors  AWS  Form  80 


6-2.  Total  number  of  briefings  (175-1, 
SAC  597-598,  AWS  28) 


M.I.S.  Report 
to  MAC  HQ 


6-3.  Number  positive  comments  on  SAC 

Form  352 _ _ _  SAC  Form  352 

Total  number  comments  received  on  SAC  Form  3 $2 

SAC  Form  352 


KRA  7:  Meet  the  needs  of  operational  requirements  and  unit 
personnel  through  good  personnel  management. 

7-1.  Percent  Officer  Efficiency  Reports,  Local  Report 

Airmen  Performance  Ratings  submitted 
on  time 

7-2.  Number  disciplinary  actions  (Letters  Local  Records, 

of  Reprimand,  Article  15's,  etc.)  Not  Tabulated 
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Sample  List 

Weather  Organization  5 

Key  Results  Areas,  Indicators  and  Data  Sources 


KRA  1:  Issue  accurate  terminal  area  forecasts. 


1-1 .  Number  incorrect  forecasts _ 

Number  incorrect  forecasts  allowed 
(Standard) 

1-2.  Number  forecast  amendments  required 
1-3.  Number  bust  reviews  required 


Local  Records 
Air  Weather 
Service  Standard 

3rd  W  W  Form  39A 

Local  Records , 
Not  Tabulated 


Number  forecasts  with  discrepancies 

in  forecast  coding _ _ _ 

Total  number  forecasts 


AWS  Form  80 
AWS  Form  80 


Number  forecasts  not  submitted  or  not 
submitted  on  time 


3rd  W  W  Form  1 


KRA  2:  Accurately  and  timely  record  and  transmit  weather 
observations  locally  and  by  long-line. 


Number  discrepancies  on  AWS  Form  10 
Total  number  Observations 


AWS  Form  80  & 
AWS  Form  1 
AWS  Form- 8ir~& 
Form  1 


Number  electrowriter  errors 
Number  observations 


AWS  Form  80 
AWS  Form  80 


Number  late  observations _ 

Standard  for  late  observations 


3rd  W  W  Form  16 
AWS  Standard 


KRA  3:  Issue  timely  and  accurate  weather  warnings 

3- 1 .  Number  warnings  issued  that  verify  ; 

Number  warnings  issued 

3-2.  Number  warnings  issued  with  desired 

lead  time _ _ 

Number  warning  that  require  lead  time  i 

3-3.  Number  bust  reviews  for  missed  warnings  ] 

(No  Lead  Time,  etc.)  1 


3rd  W  W  Form  16 
3rd  W  W  Form  16 


3rd  W  W  Form  16 
Local  Records 

Local  Records, 
Not  Tabulated 
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3-2:  By  type  injury,  Number  of  people  injured  Safety 

on  job _ _  office 

Number  assigned  record 


KRA  4:  Full  utilization  of  personnel  and  material  resources. 


4-1.  By  shift,  by  section.  Number  of  hours  on  the  AFTO 

job  (Time  x  +  1) _  Form 

Number  of  hours  on  the  349 
job  (Time  x) 


4-2 .  Actual  time  to  complete  job 
Standard  performance  time 


Locally 

maintained 

record _ 

AFR  400-1 


4-3.  Number  hours  test  equipment  in  Precision 
Measuring  Equipment  Laboratory  (PMEL) 

(Time  x  +  1) _ _ _ 

Number  hours  test  equipment  in  Precision 
Measuring  Equipment  Laboratory  (PMEL) 
(Time  x) 

4-4.  Number  of  Material  Deficiency  Reports 
(MDRs )  submitted  to  depot  that  receive 

proper  response _ 

Number  of  MDRs  to  depot 


4-5.  By  stock  number.  Number  items  cross- 

canned  (Time  x  +  1) _ _ 

By  stock  number,  Number  items  cross- 
canned  (Time  x) 


Locally 

maintained 

record 


Locally 

maintained 

file _ 

Deficiency 

analysis 

Monthly 

maintenance 

summary 


KRA  5:  Secondary  equipment  in  highest  state  of  readiness. 


5-1.  By  item.  Number  out  of  commission 
Total  number  in  inventory 


Locally 

maintained 

record 


5-2.  Number  hours  required  to  keep  equipment 

in  state  of  readiness  (Time  x  +  1) _ 

Number  hours  required  to  keep  equipment 
in  state  of  readiness  (Time  x) 


Monthly 

maintenance 

report 


5-3.  Equipment  breaking  most  frequently 


Locally 

maintained 

record 
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Sample  List 

Propulsion  Organization  6 
Key  Results  Areas,  Indicators  and  Data  Sources 


KRA  1:  Quality  power  plants  for  assigned  aircraft  at  right 
time  and  place. 


1-1.  Number  of  spare  engines  available 


Maintenance 
control  - 
local  record 


1-2.  By  cause,  Number  of  functional  check  flight  Local  daily 
(FCF)  non-releases  due  to  engine  problems  status  sheet 

(Time  x  +  1) _ _ _ 

By  cause.  Number  of  functional  check  flight 
(FCF)  non-releases  due  to  engine  problems 
(Time  x) 


1-3.  Flyable  aircraft 
Total  inventory 


Local  daily 

deviation 

sheet 


1-4.  Number  of  test  cell  rejects 
Number  of  engines  tested 


Test  cell 
reject  sheet 


1-5.  Number  of  repeat/recurring  engine  write-ups 
Number  of  one-time  engine  write-ups 


Special 
"flightline" 
form  for 
each  engine 


KRA  2:  An  effective  training  program  producing  technically 
qualified  personnel. 


2-1.  Number  of  3-levels  assigned 
Number  of  qualified  trainers 


2-2.  End-of-ccurse  test  scores 


Maintenance 
management 
information 
and  control 
system  (MMICS) 

Training 

section 

record 


KRA  3:  Safe  operation. 

3-1.  Number  of  detected  safety  violations  (DSV) 

(Time  x  +  1) _ 

Number  of  DSVs  (Time  x) 


Quality  Control 

report  and 

maintenance 

standardization 

evaluation 

program  (MSEP) 

reports 
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Locally 

maintained 

record 


5-3.  Number  of  Freedom  of  Information  Act, 

Privacy  Act,  and  AF  Form  844  requests 
processed  incorrectly  (Time  x  +  1) 

Number  of  Freedom  of  Information  Act, 

Privacy  Act,  and  AF  Form  844  requests 
processed  incorrectly  (Time  x) 

5-4.  Number  of  reports/ jobs  reaccomplished  Not  available 

due  to  improper  procedures  being  used 
by  Administration  Division  personnel 
Total  number  of  reports/jobs  done  Locally 

maintained 

record 
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KRA  3:  Morale  of  administrative  personnel. 

3-1.  Number  of  non-mission-related  conflicting 

demands  (Time  x  +  1) _ 

Number  of  non-mission-related  conflicting 
demands  (Time  x) 

3-2.  Number  of  monetary  and  non-monetary 

incentives  awarded  (Time  x  +  1) _ 

Number  of  monetary  and  non-monetary 
incentives  awarded  (Time  x) 

3-3.  Average  number  of  special  military 

details  outside  of  normal  assigned  duties 


KRA  4:  Minimize  the  cost  of  supplies  and  equipment 

4-1.  Number  of  times  pouch  mail  is  requested 
to  be  mailed  out  of  pouch  (Time  x  +  1) 

Number  of  times  pouch  mail  is  requested 
to  be  mailed  out  of  pouch  (Time  x) 

4-2.  Number  of  reams  of  obsolete  forms  reused 
for  public  use 


4-3.  Number/quality  of  equipment/supplies  not 
being  used 


4-4.  Quarterly  cost  of  administrative  supplies 

(Time  x  +  1) _ _ 

Quarterly  cost  of  administrative  supplies 
(Time  x) 


KRA  5:  Stay  abreast  of  all  current  regulations  and 

5-1.  Number  of  Operational  Readiness  Inspection, 
self-assistance,  and  in-house  write-ups 


5-2 .  Number  of  work  request  denials 
Number  of  authorized  requests 


Locally 

maintained 

records 


Locally 

maintained 

record 


Locally 

maintained 

record 


Locally 

maintained 

record 


Locally 

maintained 

record 

Locally 
maintained 
record  and 
supply  records 

Base 

financial 

record 


procedures . 

SAC  Form  210 
and  locally 
maintained 
record 

Locally 
maintain  d 
record 
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Sample  List 


Administration  Organization  5 
Key  Results  Areas,  Indicators  and  Data  Sources 


KRA  1:  Direction  and  leadership  for  base  personnel  on 
administrative  matters. 

1-1.  Number  of  telephone  inquiries  received 
and  honored  regarding  administrative 

procedures  (Time  x  +  1) _ 

Number  of  telephone  inquiries  received 
and  honored  regarding  administrative 
procedures  (Time  x) 

1-2.  Average  time  from  inquiry  to  response  Locally 

maintained 

record 

1-3.  Number  of  classes,  films,  etc.  provided  Locally 

base  personnel  (Time  x  +  1) _ maintained 

Number  of  classes,  films,  etc.  provided  record 

base  personnel  (Time  x) 

1-4.  Number  of  unsolicited  contacts  with  other  Locally 

base  offices  (other  than  staff  assistance  maintained 

units)  (Time  x  +  1) _ record 

Number  of  unsolicited  contacts  with  other 
base  offices  (other  than  staff  assistance 
units)  (Time  x) 

1-5.  Number  of  staff  assistance  visits  requested 

by  b ase  personnel  (Time  x  +  1) _ 

Number  of  staff  assistance  visits  requested 
by  base  personnel  (Time  x) 


Locally 

maintained 

record 


KRA  2:  Deliver  administrative  services  to  the  customer. 

2-1.  Number  of  pieces  of  mail  processed  per  week  Monthly  log 

Number  hours  worked  (Week  x  +  1)  and  time 

-  cards 

Number  of  pieces  of  mail  processed  per  week 
Number  hours  worked  (Week  x) 

2-2.  Average  turnaround  time  from  date  of  work  AF  Form  844 

request  to  date  of  deliveries 

2-3.  Average  time  from  date  of  work  request  AF  Form  844 

to  initiation  of  work 

2-4.  Number  of  backlogged  work  orders _ Locally 

Standard  number  of  backlogged  work  orders  maintained 

record 


As  a  resume  >f  participating  in  this  process,  how  would 
you  describe  your  current  level  of  awareness  of  the 
meaning  of  productivity  in  Air  Force  organizations? 


(Check 

one) 

1. 

Very  Low 

2. 

Low 

3. 

Moderate 

4. 

High 

5. 

Very  High 

What  other  reactions  do  you  have  to  the  process,  its 
results,  or  its  utility  for  your  organization? 


*3.3.  Use  Nominal  Group  Technique  (NGT)  to  define  and 
prioritize  KRAs 

*3.3.1.  Silent  generation 

*3.3.2.  Round-robin  listing 

*3.3.3.  Clarification 

*3.3.4.  Preliminary  vote 

*3.3.5.  Discussion  of  voting  patterns 

3.3.6.  Vote  2 

3.3.7.  Tally  results  of  vote  2 
*3.4.  Sanity  check  of  KRAs 

3.5.  Develop  chart  listing  and  describing  KRAs 
Phase  4:  Development  of  Indicators 

4.1.  Form  Team  B 
*4.2.  Orient  Team  B 

4.3.  Generate  productivity  indices  for  KRAs 

4.3.1.  Silent  generation 

4.3.2.  Round-robin  listing 

4.3.3.  Clarification 

4.3.4.  Vote  1 

4.3.5.  Discussion  of  preliminary  vote 

4.3.6.  Final  vote 

*4.3.7.  Sanity  check  of  indicators 

4.4.  Develop  chart  listing  KRAs  and  indicators 
♦Phase  5:  Review  of  Indicators  and  Data  Sources 

*5.1.  Schedule  meeting  with  target  unit  commander 
*5.2.  Review  indicators  and  determine  data  sources 
*5.3.  Develop  chart  of  KRAs,  indicators  and  data  sources 
*5.4.  Feedback  of  final  results  to  participants 
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